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ABSTRACT 

Twenty pairs of students from seven special education 
resource rooms (elementary and intermediate grades) were assessed on 
th,e structure of the instruction they received. Their, reading 
achievement was measured, three times during -the year. The • 
instructional programs for 10 experimental students were to be based 

of continuous evaluation procedures; these 
also on an accuracy of implementation scale, 
the procedures were administered and the data 
and that teachers considered the procedures 
time efficient and useful in monitoring student progress. However, 
the procedures rarely were used to evaluate instructional programs, 
make instructional changes, or monitor, the; effectiveness of ,the 
changes. No statistically significant, differences in instructional _ 
structure or achievement were found between experimental and control 
Ss. (Author/CL) ... 
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Abstract 

Twenty pairs of students from seven special education ^resource 

rooms were assessed on the structure of the instruction they received 

and their reading achievement at three times during* the year, m The 

instructional programs* for 10 experimental stutlents were to be based 

on their teachers'* use of continuous evaluation procedures; these 

students v were assessed also on an accuracy of implementation scaie. 

Results indicated that the procedures w£re administered N^nd the data 

charted appropriately^ and>that teachers considered the ^procedures 

time efficient and useful in monitoring student progress. However, 

"the procedures rarely were used to evaluate instructional programs, to 

make instructional changes, or to monitor the effectiveness of 

changes. No sta&ist.ically significant differences in instructional 

structure or achievement were, found between experimental and control 

group students. Jhe implications of non-implementation of the 
• * 

evaluation aspects .of continuous evaluation procedures are discussec|.' 



The Effects' of Training Teachers in the Use of 
Formative Evaluation in Reading^ An Experiment* l-Control)()p^ari son « 

In recent years, with the advent Of Public Law. "94-142 and 

» » 

increased public * pressure for accountability iji education, greater 
demands have -been placed on educators, Especially special educators, 
to be accountable for the quality of instructional ^decisions and the 
ways in/fthich t'hey are madl. A growing' body of evidence (Bohannon,. 
1975; Grutcher & Hofmeister, 1975; Lovittj Schaff, & Sayre,' 1970; 
White & Liberty, 1976,) suggests that formative evaluation- systems may 

r 

provide ^viable' alternatives, to the traditional pre- andj/post-testing 
approach to evaluation of academic programs. Such systems provide 
continuous feedback, to both the teacher s and student, allowing 
educators to more "closely monitor academic progress. 

luring the . past „ five years, the Institute for- Research on 
^Learnfr1g"«,Disabilities at the University of Minnesota, under federal 
contract, .-has conducted^ number of studies that focus on. developing 
and monitoring progress on \eP goals, as * is intended ifl PL°94-142. 
The goal of this research has been to determine empirically the 
effects of using formative evaluation techniques on student 
achievement in reading, spelling, and written expression. 

Earlier research in this 'area determined what measures of student 
performance would be Ideal for*use 1 in a forma£iv£ evaluation systefn. 
The search for these measures began by generating^ list of desired 
characteristics, such'as ease of administration, time efficiency/ and 
Sensitivity* to growth over time (Jenkins, Deno, & Mirkin, 197.9). The 
measures that* were not reliable or vali^, or those that were deemed 



2 • , 

less suitabte with respect to any .. of the other desired 

characteristics, were eliminated from consideration. 

r • \ 

Five reading behaviors were generated^ from a review of the^ 

literature and placed in the original pool for consideration. A 
* * ■ '* > 

series »of criterion validity studies (Deno, Mirkin, Sfajang, & Lowry, 

•*% 

1980), showecf that reading aloud from a basal reader, reading aloud 
from lists of isolated words, and guessing the words deleted from a 
reading passage (i.e.-, cloze comprehension) all related closely to 
performance on standardized tests and discriminated between program 

and grade placement. Such formative measures of reading also have 

* • * $ ' ■ - 

shown high test-retest (jr = .90) and alternate-fbrms. (rs s .89 - .92) 

reliability (Sh'inn,, 1981). " * 

Related studies focused on determining the optimal duration of 

reading measurement and the type of data to retort!.. Results ?rom 

testing one, two, and three-minute durations indicated that reading 

proficiency can be ifidexed validly within* one minute and' that co^rept 

performance is a more va*lid measure of reading proficiency than error 

performance (Deno et al., 1980). « . 

Previous studies also assessed the sensitivity, of two reading^ 

measures, reading isolated word lists and reddiffg aloud from a basal 

it ' 

reader. Both reading measures were found to be sensitive to changes * 
within each grade, level from fall to spring and~across grade levels 
(Marston, Lowry, Deno, & Mirkin, 1981). However, reading aloud from a 
basal reader wafc chosen as the* optimal generic measure in reading 
because it produced a broader range of scores ttfan isolated words, 
related somewhat more closely to comprehension,- a/id required Tittle 



teacher preparation. ' , 
Rationale \ ^ ' , ' ■ , % 

Given that one-minute . timed samples of reading from the 

' x V / ' * ' • • - ' • / 

curriculum hVve, been -shown to be reliable and val/M measures of 

' *' 

reading growth, there remained the need to test the practicality of 
such measures*., and .any effects teaeheY use of such measures/mi^lfc have, 
on student lichjevepient over time. A number of questions related to 
these issues wore pose'i 'in the current study. • " ' ' ' ".» ' 

*\ ■ c * 

First, can teachers learn to use the' measurement system and will 
they find* it practical and .time-eff icfent? Such measures, however 
reliable .and,valid, /Vnll come into widespread use.fn the classroom 
vdnly .if 1 teaclters f»ind them to be a sayings*, rather than an added 
burden, .in time-'spent in evaluation. " 

Second,- will the recArd of student growth presented in charts -be 

i . 

perceived by teachers of special education as an actual representation 
bf,fthe student's, progress in reading? Again, the extent to which 
teachers believe the. information they obtain reflects actual growth 
will determine the extent to which/the instcumeqt is /ised ,1n tjie 
classroom. • 

Third, once the measurement system is implemented, will resource 
room teachers use the information it provides to mote closely monitor 
£nd change the educatymal program of the student? One of the major 
chantages of sue+r a systenr^is* that it .."allows, for continuous 
evaluation- of, the instructional program. Thus, it 1s*. critical * that 

m 

**the* information provided ^by the system be used. * 

Only if all these questions are answered affirmatively is it 



possible, to - examine the questions concerning the effectiveness of the 

system. Two- questions were investigated 'concerning"" the efficacy of 

the 'measures. Given tha£^*elchers can learn to use a -formative 

evaluation system for both measurement and evaluation/ will the use of 

such'^r-ocedures have, a direct effect on student : achievement? One 

Would expect that the frequent modifications "in the instructional plan 

m$de passible by continuous feedback wquVd lead to an educational 

program more sensitive to individual °r\eed$ and thus more cmducive- .to 

gro^h 1a, reading, Second, will the use j of such measures^ h r ave an 

effect' on' the structure of^the learning environment provided to the 

student? Because a formative evaluation - system provides .coritinuous 

information about the need for program changes, dne might expect the 

use of such a system to result »in- a more highly' structured 'learning 

v . ' f ~ ■ 1 

environment. ] - » 

# • « \ 

- ■ ' l ■ . ^ 'Method 

' * 

The subjects were 36 elementary and 4 intermediate students in 

s * \ ' - 
special edacaticfo* -resource 1 classrooms in a rural . educational 

cooperative, representing 20 experimental-control matched pairs. The 

'mean grade level was 3. : 8, with 18 students in grades 1 through' 3, 18 

students* in s grades : 4-- through 6, and 4' students in grades 7 and 8. All 

* ' * • 

subjects were functioning dramatically belSw their peers in reading. 
Thirty of the subjects were boys, tea were^girls. All subjects" vfere, 
, studied in the resource room setting; their teachers were seven 
special education resource teachers' whose experience ranged from two 
to six -years, with a median of three years teaching special education. 



The original pool included, 46 o subjects (23 matched pairs) in nine 
resource rooms; however, two of the teachers^ failed to implement the 

continuous evaluation procedures and the, three pairs in tjiese 

. . . \ 

classrooms" thus wftre dropped from the study. * 
Procedures » _ 

The resource room teachers were trained in • the use of the 
measurement procedures during a series of three* half-day. workshops at 
the beginning o¥ the school year. Training'Was based on the manual, 
Procedures -to Develop and Monitor Progress on IEP Goals (Mirkin, Oeno, 
'fuchs, Wesstn; Tindal, Mars ton, & Kuehjnle, 1981)-. *The teachers 
Continued to use the measures over the entire school year. Visits by 
observers *in December, February, and M&y, and freguent phone contacts, 
'provided feedback to the teachers on the accuracy of their 
implementation^ the measures., 

Daily measurement consisted of one-minute timed samples cff 

reading from the- stude/it's curriculum; Both words correct and 

I 

incorrect were scoYed* and charted on equal interval charts. Based on 

O . 4 

the results of previous research, the placement level for, testing, 

* 

which also became the baseline, was'set at a criteria of 20-29 words, 
1 „ - • ■ ** 

per minute for grades 1 and 2, and 30-39 words per minute for grades 3 

thfougfi 87~ ~ . 

Teachers were instructed to write IEP long-range goals using both 
the entry level criteria and a desired year-end mastery criteria, 
usually 70 words correct per minute- with no more than 7 errors. The 
formula used in. writing the long-range goal is shown in Rigure 1. 
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Insert Figure 1 about here 



— r — --- 

Short-term objectives were based on the 1'ong-range goals (LRG). 
In order v to compute the short-term objective, teachers 'first 
subtracted the baseline level of performance from the criterion level 
listed in the LRG. Dividing this difference by the number of weeks 
necessary until the annual review, they arrived at the number of words 
per week gain necessary to meet the long-range goal criteria, the 
format used for writing short-term objectives is given in Figure 2. Jj 



Insert Figure 2 about here 



In addition, the teachers also were trained, at the beginning of 
.the .year, and again at m>d-year, in the use " of the measurement 
procedures for^4valuatfon of the instructional program. Irv order to 
monitor student growth, the baseline reading levftj and the long-range 
goal wefe conaetjed by an aimline that showed the students 1 desired 
progress. Every seven data points, the teachers were to t moo it or -/ 
student growth~"by means of the split-middle or quarter-intersect 
method (White & Haring, 1980). An' example is given in Figure^. If * 
,the student was progressing at a rate equivalent to or greater 'than 
• that indicated by the aimline, the instructional program was 
continued; if the projected rate of growth was less than that 
indicated by the aimline, teachers were directed to make a substantial 

change in the student's program. \ 

* " car 



li 



'insert Figure 3 about here 



* Measures p 

+* 

Four measures were used in collecting data: one each for 
implementation and structure, and two for achievement.. The structure 
of the learning environment was assessed by means of the Structure of 
Instruction Rating Scale for both experimental and control subjects. 
Degree of implementation of the continuous evaluation measures—the 
treatment for the experimental subjects— was assessed using the 
Accuracy of Implementation Rating Scale. Achievement; measures for 
both' experimental and control subjects consisted of timed samples from 
three third grade passages, and four subtests of the Stanford 
Diagnostic Reading Test (SDRT). -The three timed samples were 
collected three times during the year, in November, soon after the 
teachers had begun implementing the measures, in February, and, in May, 
at the end of the school -year. * The Stanford^ Diagnostic Reading Test 
was administered in May. Qescriptioris of the measures follow. 

Structure of instcuction rating scale (SI£S). The Structure bf 
Instruction Rating, Scale (SIRS) was designed *to measure the degree of 
structure of the instructional lesson that q student received* in this 
case in reading. The variables chosen 'Tor inclusion on the SIRS were 
gathered from current literature on instruction and student academic 
achievement (cf. Stevens & Rosenshine, 1981). A list of the variables 
and their operational definitions can be found in Appendix A. 

The SIRS consists of 12 five-p*oint bipolar rating scales. A 
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rating. of 1 is low for the variable and 5 is high. Observers, trained 
by videotape to a criteria of .80-. 90 inter-rater agreement, rate all 
variables Qn the bas,is of strict definitions at Jthe end of a 20 minute 

i 

observation period. The focus of each observation period for the SIRS 
is on the instructional environment for one student at a time. For 
the current study, nine research assistants w£re trained as observers 
and reached an inter-rater agreement level of .92 before actually 
observing in the classroom. 

The reliability of the SIRS was assessed by means of Coefficient 
Alpha, a measure of internal consistency. For, a sample of 70 students 
observed in November, the average inter-item correlation was .37, 
resulting in an alpha of .86. Thus, the SIRS sefems to have a high 
degree of rel iability as indexed bjMfi^ux^sjDf ■ homogeneity. 

Factor analysis of the 12 variables on the sfRS>ev^aled that 9 of 
the 12 represented one factor. Three variables-^IndependenTl^^ 
Positive Consequences, and Silent Practice on Outcome Behavior— were 
not measuring the same factor. Thus,, the nine variables were utilized 

i 

in the data analyses as one factor and the other three variables were 

«t 

analyzed separately. 

Accuracy of implementation rating scale (AIRS) . The Accuracy of 

Implementation Rating Scale (AIRS) is an instrument that was developed 
°in conjunction with the manual Procedures to Develop and Monitor 

Progress on IEP Goals (Mirkin et al., 1981). The AIRS is designed to 
"provide a format by which to monitor the implementation of the 

procedures described in the manual. The AIRS consists of 12 items 

that are rated on a 1 to 5 scale, ! being the lowest implementation 



score and 5 being complete and accurate implementation.. A copy Qf the 
scale and variable definitions is included in Appendix B. 

Parts of the scale require direct observation whereas other 'items 
on the checklist can be monitored by inspection of student reading 
graphs and by reading IEP forms. Items 1 and 2 of the AIRS, which 
require direct observation, deal with the accuracy of administration 
of the measurement and selection of the stimulus materials. The nine 
observers trained in the use of the SIRS were trained concurrently on 
these two items, and observed and rated these variables in the 
classroom during the same period as they completed the SIRS. Items 
3-12 of the AIRS require inspection of various written documents. 
Specifically, the rater examines the following documents, for^ -each 
student: (a) the IEP, which should specify the long-range goal and 
short-term objective in reading; (b) the reading graph; (c) the 
instructional plan for reading; and (d) the record of changes made in 
the instructional plan in reading. Factors included in^ items 3-12 
pertain to accuracy of establishing: (a) the appropriate measurement 
level; (b) an adequate^ baseline; (c) an accurate long-range goal and 
short-term objective;- (d) a detailed ,grapli; (e) a complete, 

instructional pVogram; and (f) the aimline. These items also focus on 

\ ■ 

the timing of instructional changes as well as the* types of changes 

\ 

made. Items 3-l£ were rated by research assistants. Frequent checks 
among the four research assistants completing items 3-12 assured 
inter-rater agreement of at least .80. 

Reliability of the AIRS was assessed ,by means of the Cronbach's 
Alpha internal consistency measure. The average inter-item 



u , 7 

correlation was .12, resulting in dfh alpha of .62, somewhat lower than 

0 "... * / 1 

-the SIRS, but adequate £)or researcf^purposes. 

Results , * 

implementation 

The" mean ratings for each variable on the Accuracy of 
Implementation Rating Scale for all three rounds of data collection 
ar^ shown in Table 1. For purposes of analyses, the scale was divided* 
•Into three subscales: initial measurement (procedures necessary for 

. 7 

setting up the measurement' system and thereafter modified only if in^ 
error), ongeing measurement (procedures performed- -frequently that 
provide ongoing • data), and- evaluation (processes that » involve 
evaluating instructior\aljiclbgress and making changes). Scores Jn the 
table represent the mean of a five-point rating scale. . 



Insert* Table 1- about "here 



As can be seen in the table, relatively high scores were obtained 
in both initial and oncfbing measurement. Initial measurement 
procedures such as setting up the long-range goal and short-term 
objective reached hicfh levels soon after implementation and maintained- 
an average rating of approximately 3.9 for all three rounds of data 
collection. Accuracy of ongoing measurement tasks' initially was rated 
somewhat / lov^r (X* - 3PL66) ' by observers, but increased by round, two, 

* f. * - 

and procedures such* as administration of the measurement task were the 
most highly rated by the end of the year (X a 4.24). 

The ratings for procedures used in evaluating the data and making 
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instructional changes- differed from the other two sets, however. Even 
though a mid-year ^training workshop on using the measurement system to 
make instructional changes was conducted "between data collections two 
and three, the mean ratings of evaluation procedures tended to remain 
below those of other variables (7 = 3.6). Higher ratings on the 
variables relating to substantial and clear changes indicate that when 
changes were made they were in accordance with the prescriptions of 
the system. However, only<*th.ose experimental subjects for whom 
changes were made could be rated on these variables, and it is clear 
from the low number of cases for both these variables that this 
represents a small subset of the subjects. ■ ^ 

The low frequency of instructional changes implemented was 
striking. For the 20 students receiving the experimental treatment, 

v \ 

only 27 changes were made in instructional programs, meaning that the 
average number of changes made in the instructional plan during the 
entire school year for any individual * was 1.4. As is shown in Table 
2, no changes in the instructional plan were made for seven of the 

4 

experimental subjects, and for only three of the subjects were more 

than two instructional changes made during the course of the school 

year. tt is clear from these data that a potentially .important 

component of the experimental treatment, evaluation of changes in 

reading instruction, was implemented only partially by the resource 

room teachers in this^ study. In fact, making changes according to the 

data utilization rule (i.e., timing*of*instructiona1 changes) received 

one of the lowest mean ratings. (3.39). In summary, teachers. made few 

t? - ■ *■ 

changes in student programs and seldom when the data called for such 

• 16 
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changes. 



Insert Table 2 about here 

Structure of Instruction • 

The mean ratings for each variable* on * the Structure of- 
Instruction Rating Scaje are^ reported in Table 3* Classroom 
structure, as measured by the scale, for the most part remained stable 
over time. Among tne nine variables analyzed as one factor, the 
moderate to high ratings at all three points in time indicate that 
these aspects of classroom structure are relatively well established 

) , 

in most resource rooms. It is apparent, however, that the elements of 
structure represented by Independent Practice, Positive Consequences, 
and Silent Practice were much less a pat*t of the classrooms studied. 
Considering the importance placed on token economies and point systems 
for special populations (Kazdin, 1977; O'Leary, Drabman, & Kass, 1973; 
Phillips, Phillips, Fixsen, & Wolf , 1971; Staats &1utterf ield, 1965), 
the almost, total absence of such systems is # especially surprising. 
The average rating on Positive Consequences in the May observations (X 
= 1.13) means that, of tye 38 students observed, only two f were 
observed receiving any form of token economy in their resource room. 

\ Insert Table 3 about here 

Although one would not expect any significant differences to 
emerge between experimental and control subjects on the SIRS since. the 



13 

treatment was not fully implemented for Experimental subjects, mean 
differences for experimental and control subjects /are reported in 
Table 4. None of the differences were statistical ly signif icanO 
Still, it is worthwhile to note that differences 'in sample means that) 
do appear between experimental and control subjects are in °favor of 
the experimental subjects on 9 of 12 rated variables. 



Insert Table 4 atibut here 



Achievement 

The achievement measures showed high degrees of correlation- among 
themselves. The high correlations, among the third grade passages (r = 
.61 to .93, with the majority of coefficients in the high .70s and 
.80s) suggest that £he passages possess a high degree of alternate- 
forms reliability. The correlations ranging from .47 to .68 between 
the passages and* the Stanford subtests reaffirm £he validity of the 
passage data for purposes of measuring reading growth. 

Overall descriptive data for the achievement measures are 
presented in Tables 5 and. 6. As 'can be seen in Table 5, each of the 
reading passages showed performance /gains over time, although passage 

3 proved less sensitive to growth "than passage 1 or 2. The average 

' / 

percentile rank for the Structural Analysis subtest of the SORT was 
40.5 (see Table 6); the average percentile rank for the Comprehension 
subtest of the SORT was 32.1. To 'control /or ■ grade differences, Z- 
transformations were performed on all achievement data prior to 
analysis. 
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Insert Tables 5 and 6 about here 

Again, siFTce the evaluation procedures were not. fully implemented 
for the experimental subjects, one must be cautious in interpreting 
the results of- experimental-control comparisons. Nevertheless the 
pattern of experimental-control achievement differences (see Tables 7 
and 8) is consistent with that obtained for structure. Although none 
of the mean differences reach the level required for statistical 
significance, the mean differences for passage data showed gains in 
favor of the experimental subjects, and three of the four SDRT 
subtests that exhibited mean differences at the end of the year were 
in favor of the experimental subjects'. \ 



Insert Tables 7 and 8 about here 



Useab ility ' '" . " 

At the end of the ye^r,. the resource room teachers who 
participated in the current study were interviewed regarding thejr 
reactions to the' continuous evaluation procedures they implemented. 
Preliminary analysis of the interview data indicated that, in general, 
the teachers reacted positively to the use of formative evaluation. 
All but one of the teachers interviewed felt that the timed reading 
sample represented a good measure of reading proficiency. In general, 
they believed that the slope of the graph reflected the actual rate of 
reading growth, After a year, of measuring, none expressed feelings 
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that- the procedures were too time-consuming. V 
Other benefits \lfere noted by the teachers as well. One stated 

fhat, "the data was more useful to me and the classroom teacher.."' 

; 

Another stressed the motivational aspects of the system* stating "It 
greatly improved the students 1 confidence which spurred him/her^^ 
i mprove ; " Others noted i mpro ved read i ng performance i n even those 
students who -failed to reach their-reading goaj. The only reservation 

c. . • 

i * 

about the continuous evaluation system concerned its relationship to 
comprehension; one teacher felt that at least one of her students' 
"improved in reading words, but not in comprehension." 

Discussion £ 
The current investigation focused on a number of research 
questions relating to the practicality 'and efficacy of formative 
evaluation systems in reading in the resource room. Can teachers 
learn to use such a measurement system, and will they find it 
vpractical, time efficient, and a believable recor^ of student^growth? 
Will they use "the continuous information provided by, such a system in 
order to make frequent changes in the educational plan, and monitor 
the effectiveness of such changes? Does such a system of measurement 
and evaluation, once implemented, have an effect on the structure Qf 
instruction or on reading achievement? 

Ratings of the accuracy of implementation indicated that teachers 
can learn to administer timed reading samples and chart the results in 
order to provide a^cbntinuous record of student growth ins reading. A 
majority of the ^teachers involved in the project felt that such 
measures were time efficient 'and s helpful .in monitoring student 



progress. In addition, all of the teachers "had confidence by" the end 
of' the year that th4 student's growth (or laclf thereof) in reading was 

accurately reflected \in his pr, her graph, Thus the teachers' can- learn 

\ 

to use the measurement system and find it both practical and ,t.ime 

efficient in the classroom. 

t . ■ ' - ■" 

Althoug% able to accurately measure student>eading progress, the 

resource room teachers rarely utilized the procedures to evalpate the 

instructional program, make instructional change^, and monitor the 

effectiveness of' these changes.' Ratings of evaluation procedures 

tended td be lower than either initial or ongoing measurement 

variables. More importantly, very few changes were made in the 

instructional programs of the experimental subjects. * Thus, for , all 

• * 
intents .and purposes, the gdal that teachers use the measurement to 

direct* and test changes in student programs was only partially- 

realized. ■ 3 * , * * 

Why teachers can master a measurement system but not utilize the 
data for educational planning «and I ; change, is ^a matter for some 
speculation. One explanation is that the techniqal requirements of 
the evaluation procedures are more demanding thai\ thq initial or 
ongoing measurement procedures. Teachers must learo to compute slopes 

in time series data and make judgments regarding program 

* -■ 

effectiveness. This requires not only new technical' skills, but is 
also time consuming. * /* 

A second explanation may lie in the nature of the demands of the 
measurement and evaluation components. • While changes in the 
procedures used for assessment may be relatively easy for most 

. .2.1 



teachers to make, .the evaluation procedures .prescribed by continuous 
evaluation may require changes in teaching procedures that teachers 

are unable or unwilling to make in ttie space of a year. It may well 

. *+ - ' 
be tfrat special education teachers, accustomed to making few changes 

in '"educational .plans, find making ■ frequent changes 'in the 

instructional plan based on graphed data to be in conflict with their 

current mode of teaching, further, the teachers may weM be doing all 

that they know how to do to teach reading and may not have a 

repertoire of alternative tactics ^rom whicH ^ to draw. Finally, few 

prompts or reinforcers^may be available for making changes, 'in fact, 

making changes may increa$e^work— a punishment contingerfcy, *' ' ' . ■ 

\ ■ > . : : 

While it would' be unrealistic to expect improved achievement with 

. : ' °« / ' 

the experimental treatment only -parti ally in effect, the data provide 

some basis for the b<?Hef that an appropriately implemented continuous 

evaluation system .could have some effect on both achievement and 

program structure/ Although none _of the experimental-control 

differences were reliable, most of the mean differences favored the 

experimental condition. - Such results provide a hopeful sign that 

continued implementation of such a'^stem, with special attention paid 

to its evaluative components-* might ° indeed accelerate the rat6 of 

0 

learning to read,- Alsd, the reports of tffe resource teachers support 
the technical adequacy and the useability of the measures in the 
classroom. 

The results of this study corroborate earlier findings indicating 
that measuring and graphing a- student's performance do not ensure that 
those data will be used to make instructional decisions, They'suggest 



2 



5 

mm 



thai further .study is necessary to determine how best to ensure that 
teachers will use the data in order to adjust instructional programs 
and Maximize growth toward goals*in reading. ~ \ 
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Table 1 . \ 

Accuracy of' Implementation Rating Scale: Mean *Rati>gi; for Experimental Subjects 3 D - 



Q 







N 


• 






N 






' „ ■ * November ; 


of cases 


February 


ofr'cases 


•May 


o,f cases 






initial neasj*ren»enL o 






* * 


" s ' * 






— | 




sampling tor instructional Level 


o.bb 


(20; ■ 


p . / b 




o Jin * 


120) 


* 0.0/ ^ 


% 


easel i ne - ^ 


9 o or* 

3.80 


(20) 


3 5Q 


(20) 


3,60 


do) 


3.63 


\ 


A iml ino 

Him i i ne • , v , 


hi a on 

,4.o0 . 


(20) 


v - 4, 55 - 


(20)' 


4 50 


(20) 


-4 62 V 




0 Lona#Ranae Goal N 


4 10 


(20) ' 








(20) 


4 13 




Short-Term Objective ^ 


4.00- 


, (20) 


4.20 


(20) ' 


4.40 


(20) 


4.* 20 - 




* * 

Instructional Plan * ' 


3.30 v 


(20) • 


3.35 - 


(20) 


3.45 


J20) 


3.33* 




r 

Mean 


3'. 93 

* ■ » 




3*. 90 




• 3.93 ' 




3.92^ 




ungoing measuremenTT^ 






* 












Admi n i sterl na thp Mpa^urpmpnt Ta^k 

*» v»Ml III 1 J wLI 'I'M w IIC IICU JUI CIIICII u 1 CJ J i\ 


3 73 


(19) 


4 50 


(20) 


4 71 


h7) 


4.31 ' 


- 

> 


Selecting the* Stimulus Material 


3.95 


(19) 


4.50 


(20) 


4 94 


(17) 


4.46 




^ ^Graph Set-up 


3.30 


(20) 


3.30 


(20) 


3.30 


(W) 


3.30 




Mean 


3.66 ' 


r 


- *.10 




4.24 




4.02 , 




Evaluation 




• 


* 








' ' --• 




Timing of Instructional' Changes 


V 3.33 


• (15) • 


3.25 


(20) 


3.6Q 


- (20), 


3.39 




* 

Substantial* Changes 


* 4.20 


:(2) 


3.33 


(9) 


3.88 


• (8) •: 


3.90 




Clear Change - ' 


3.50 


(2) 


3.77 


M 


4.00 


(8) 


3.57 




Mean 


3.78 




3.45. ■ 




3.83 




3.62 - 




.AIRS Scale Mean 


r 








• 








All Variables 


3.79 




3.82 




4.00 




3 '.85 - 



a, 



Ratings are on a scale from 1 (low) to 5 (high). 
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Table 2 

Number of Changes Made in the Instructional Program 
During the School Year 



No One Two Three 
Changes Change Changes Changes 



No. of Experimental 
Subjects 



28 ' 



Table 3 
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Structure of Instruction Rating Scale: Mean Ratings for Experimental Subjects < 







November 


(N) 


February 


(N) 


May 


(N) 


X 




Instructional Grouping 


3.43 


(37) 


2.95 


(40) 


'3.29 • 


C38) 


^3722 




Teacher-Directed Learning 


4.08 


(37) 


4.24 


(38) 


4.08 


(38) 


4.13 


* 


Art 1 I/O' flra/lomir Doc nnnrli* nn 
MVfLIVc MLdUcitML KcSpunuiny 


H • 1 D 










wo/ 






Demonstration and Prompting 


3.24 


(37) 


3.55 


(40) 


3.50 


(38) 


3.43 




Controlled Practice 


3.57 


(37)' 


3.26 


(39) 


3.43 


(37) 


3.42 


> 


Frequency of Correct Answers 


3.78 


(36) 


• 4.44 


(36) 


4.23 


(35) 


4.15 




Corrections 


4.03 


(36) 


4.51 


(37) 


4.31 


(36) 


4.28 




Pacing 


3.57 


(37) 


3.59 


(39) 


3.72 


(36) 


* 3.63 




Oral Practice 






2.23 


(40) 


2.90 


(38) 


2.57 




M^an 


3.73 




. 3.66 




3.71 




3,66 




Independent Practice 


2.17 


(18) 


1.41 


(22) 


1.56 


to) 


1 .71 




Positive Consequences 


1.62 


(37) 


1.51 


(40) 


1.13 


(38\ 


1.42 




Silent Practice 






1.98 


(40) 


1.92 


(38) 


1.95 




Mean 


1.90 




1.63 




1.54 




1.69 




a ftatings are on a sqale from 1 


(low) to 5 (high) . 
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Table 4 

Structure ^of Instruction Rating Scale: Experimental 



Control Means 



November 
Experimental Control 



February 
Experimental Control 



May . 
Experimental # Control 



Instructional Grouping 


3.76 


3.06 


2. 


95 


2. 


95 


3.39 s 


3.11 


Teacher-Directed Learning 


4.29 


3 .82 • 


4. 


50 


4. 


06 


. ' 4.28 


3.89 


Active Academic Responding 


4\18 


. 4.VB. 


4. 


15 


4. 


15 


4.22 


3/61 


Demonstration and Prompting 


3.18 


3.24 


3. 


7ft 


3. 


40 


3. "56 


3.33 


Control! ed Practice 


3.47- 


3.47 ft 


3: 


05 


3. 


47 


3.47 


3.29 


Frequency of Correct Answers 


3.69 


3.81 • 


" 4. 


53 


4. 


35 


4/38 


.4.13 


Corrections 


3.94 


3.94 


4. 


71 


4. 


59 


4°: 35 


'4.17 


Pacing * 6 - e 


■ 3.64 . 


3.4T 


r. 


74 


3. 


58 


3.93 


3.44 


Or^l Practice 






2. 


15 


2. 


3(1 


3.00 


2.89 


Mean t , 


3.77 


* 3.61- 


3. 


64 


3. 


54 


3.77 


3.46 


SIRS Rating 




o 














Independent Practice 


2.00 


; 2. do 


1. 


17 • 


. 1. 


67 


1.60 


1.60 


Positive Consequences 


1 .76 


r.53 , ' 


1. 


20 


1 . 


10 


1.11 


1.17 


Silent Practice 




>* 


* * 1. 


95 


2. 


00 


1.89 


* 2.00 


a 
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1 


c 
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Table 5 

• - 


f 








Third Grade Passage Data: 

* * 


Means and Standard 


Deviations 


for All 


Subjects 


* 


November 
Mean? (SD) 


, February 
Mean (SD) U 


May 

r t Mean (SD) 


Words Read Correctly in One Minute 














Passage 1 


32.74 


(24.4G) 


44.10 


(25.49) 


62.65 


(34.37) 


Passage 2 


32.28 


(22.67) 


45.92 


(26.98) 


65.83 


(34.27) 


Passage 3 , 


23.51 


(18.26) 


32.38* 


(19.61) 


47.98 


(28.00) 


Mean 


29.51 


(21.26) 


40.80 

• 


(|3.23) 


58.82 


(31.60) 


Words Read Incorrectly in One Minute 














Passage 1 


8.86 


(4.07F, 


8.34 


(4.92) 


,'6.30 


(4.95) 


Passage 2 v. " ^ - " * 


8.45 


(3.69) 


8.21 


(5.31) 


, 6.23 


(5.45) 


Passage 3 * 


9.93 


(5.09) 


8.90 


(4.34) 


7.80 


(5.50) 


> ,., 

Mean* 


9.08 


4^54l_ , 


8.48 


(4.45) 


6.78 


(4.96) / 


. «p 

J * m * 
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Table 6 

Stanford Diagnostic Reading Test: Means and Standard Deviations 

for All Subjects by Grade 



Grade N* Mean (Standard Deviation) Mean % Rank 



Structural Analysis Raw Score 
(Word Division & Word. Blending) 



1 


19.50, 


(2.12) ■ : " 


* 


2 > 


28.75. 


(10.00) 


* 


3 • 


40.13 


(10.26) 


48.0 


4 


39.60 


(10.61) 


31 .8 


5 


52.00 


(4.97) 


47.3 


6 


44.00 


(18.22) 


* 


7 


57.67 


(1.15) 


* 


8 


56.00 


[one case] . 


* 



Total Comprehension Raw Score 
(Literal & Inferential Comprehension 

(5.66) * 

(8.96) * 

. (12.20) 39.5 

"(1 3.23) -29.3 
(3.30) " 24.S 

(20.70) * 

•(5.03> * 

[one case] * 

* ■ 
-*' Norms not available. 



1 ' 


18.00 


2 


27.38 


3 


36.50 


4 


38.70 


5 


46.75 


6 


47.00 


7 


52.67 , 


8 


57.00 
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Table 7 

Third Grade Passage Data: Experimental - Control Mean Comparisons 





November 
Experimental 


Control 


February 
Experimental 


Control 


May 

Experimental 


Control 


Words Correct Per Minute 














Passage 1 


. 32.84 


30.89 


45.42 


41 .37 


64.90 


rr\ *A t\ 

,60.40 


Pass,age 2 * 


32.05 


31 .05 


46.37 


A A t r\ 

44.10 


67.50 


C A 1C 

64.15 


Passage '3 


23.11 


22.94 


32.95 


30.37 


48.70 


A 1 OP 

47,25. 


Mean 


29.33 


on oa 

28.29 


41.58 . 


38.61 


60.37 


57.27 


r 

Errors in One Minute 














Passage! 


" 8.14 > 


9.57 




10.00 


5,95 


6.65 


Passage 2 


8.14 


8.78 


7.50 


9.43 


6.25 


6.20 


Passage 3 , 


10.29 


9.79 


8. -50 


9.71 


7.10 


8.50 


Mean I 


8.86 


9.38; 


7.74 


9.71 


6.43 


7.11 




< 
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Table 8 ' 

Stanford Diagnostic Reading Test: • 
Experimental -Ciontro^ Mean Comparisons/ 



Subtest • 


Experimental 


May 

Control 


- 


Word Division 


20.75 


19.70" 




Word Blending 


20.55 


18.95 




Structural Analysis 


41.30 


38.65 




Literal Comprehension 


20.85 


20.65 




Inferential Comprehension 


17.60 


17.80 ' 




Total Comprehension 


37.75 


38.45 





a Entries in tables are mean raw scores. 
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Condition 



Behavior 



Criteria 



LRG: 



In * ■ weeks, when 

(total # weeks) 
presented with stories from 



Level l 



(reading series), 



student will 
read aloud 



at the rate of 50 

wpm or better 

5 or fewer errors*. 



Figure 1. Format for Long-Range Goal: Reading 



r 



Condition 


Behavior 

V 


Criteria 


Each successive week, when 


' j, 

student will 


at an average 


presented with a random 


read aloud 


increase of 


selection from Level 






(#) 




(repeated-actual 


of 




performance/total # 


(reading series), 




weeks) words correct/ 






minute and no increase 






in errors. 



Figure 2. Format for Short-Term Objective: Reading" 



Student: • •Jp 



Or/Age S//o Schslfc/fy Craue f*teM%. tfukltr Academic Area: "£e»Jn 




M M 

ok 

MiAl 



M 



M M *M M M M 

' <th' ik/ /<L"- ik* *ib Jb uk 



M M M M M M 

<i-L ^ <ik J* Ji? 



M M M M M M M 

J; *1l Ai ik ik ik *k 



Figure 3. Using the split-middle technique to monitor student progress, 
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Appendix A 



School : 



■ 

Structure of Instruction Rating Scale (SIRS) 



Date: 



Student :_ 
Teacher: 



Observer: 



Number of Students in Group:. 



Number of observations" prior to rating; v ^ 
Time observation begins: • " Time observation ends:. 

Time allocated to reading instruction per day:_ 

.Currirulum used for instruction: Publishe r 



Level 



Series * 
I ns-t ructions * 

Circle the number that accurately reflects your rating for each 
variable.' Only one number may be circled per variable. If you are 
unable to evaluate a certain variable, mark N/A (not applicable) next 
to the left-hand column. 



1. 


Instructional Grouping 


1 


2 < 


3 




5 


2. 


Teacher-directed Learning 


.- I m 


2 


3 




5 


3- 


Actjve Academic Responding 


1 


2 


3 




5 


4. 


Demonstration/Prompting 


1 


2 


3 




5 




'Controlled Practice * 


1 


2 


3 




5 


6. 


Frequency of Correct Answers 


1 


2 


3 


/; 


5 


7. 


Independent Practice 


1 „ 


2 


3 • 




5 


8. 


Corrections 


1 


2 


3 




5 


9. 


Positive Consequences 


. 1 


2 


3 




5 


10, 


Pacing 


1 


2 


' '3 • 




5 


11. 


Oral Practice on Outcome 
Behavior 


1 


2 . 


3 




. 5, 


12. 


Silent Practice on Outcome 
Behavior 


1 


2 


3 


4 


5 



A- 2 

SIRS 

Operational Definitions Codebook e-*n 
Instructional Grouping 

5 - 90% or more of the instruction* this student receives from the' 
teacher is on an individual basis. 

, f 
1 - 105? or less of Tihe instruction this student receives from the 
teacher is on an individual basis \ j ^- 

Teacher-Directed Learning * 

.„„..-.-.. % ■■ * 

5 - Student's instruction is extremely organized, businesslike, 
and teacher i\s firm in direction and control of activities. 
For example, student is presented with questions, student 
has material to cover., etc. 

* *i 
1 - Student's instruction is casually organized and very spon- 
taneous.. Teacher is not committed to having the student work 
on a particular set of material. Instructional materials do 
not determine what activities student engages in and the les- 
sons change according to problems or mood of this student. 

Active Academic Responding 

• . ■* 

5 - The student is actively practicing the academic skills to be 
learned more than 75% of the time observed. 'Specifically, the 
student is engaged in oral or written responding to teacher 
questions or written material, e.g., reading aloud, answering 
question^, writing, or computing. Student- rarely is involved 
in, non-academf c conversations with teacher or othe£ students - 
Attending to the lesson without responding, such as sitting, 
looking, listening, and/or following along in a book does not 
apply. The student must make an* active, written or oral 
response. 

» - » * 

1 - The student is actively practicing the skills to be learned 
. less than 10% of the time observed. Instructional lessons 
may be interrupted or shortened to include "process" and other 
non-academic activities, e.g.", clarifying feelings, opinions', 
and working on arts and crafts. 0 

Demonstration and Prompting ' ' , . y 

5 - Appropriate steps of the desired behavior' -to be performed /re 
demonstrated for the student. Student is given an opportunity 
to practice the step(s) as teacher provides prompts for correct 
behavior that approximates or aclrives desired response. 



1 - Teacher attempts to teach the Student a behavior without using 
demonstration and prompting techniques. 



SIRS 

5. Controlled Practice 



5 - Student's practice of material is actively controlled by 
teacher whp frequently asks questions to clarify that the 
student understands what has just been demonstrated. Ques- 
tions are convergent ( single factual answer ) and the stu- 
dent's answers consistently follow the questions and are 
given teacher feedback. 

1 - Student is rarely questioned by teacher following demonstra- 
tion of new materials. Questions are more divergent (open- 
ended, several interpretations) than convergent (single factual 
answer) . .Student' s response is not .consistently followed by , 
teacher feedback. The type of questions are such that several 
answers are acceptable, i.e., questions are abstract or am- 
biguous . v . - 



Examples: 

If during an or<al reading session: 

a) trie teacher frequently attempts to clarify the material with* 
convergent questions ("what color hat was John wearing?"), a 
5 would be recorded. 

b) the teacher asks few questions, most of which are divergent , 
("What do you think this mpans? 11 ), a 1 would be recorded. 

c) . the teacher asks few convergent questions or many divergent 

questions, the appropriate" rating would be a 3. 

6. Frequency of Correct Answers 

5 - Academic lessons are conducted in such a way that the difficulty 
of the material allows the student to achieve mean accuracy 
of 80% or higher. 

1 - Academic material is difficult for student, component steps 
are large or unsequenced, and mean accuracy for student is 
less than 55%. ' * - 

(Note: Vf the student has no opportunity for oral oiLwritten response 
* during the observational period, item 6 would be rated N/A - 
not applicable, while items 3 and 5 would most likely be 
rated l)^ 

7 . Independent Practice * 

- When engaged in independent seatwork, the student frequently is 
monitored by the teacher who assists, clarifies, and praises 
the student for academic engaged tasks, 

(Note: Independent seatwork is defined here as a student working on an 
assigned task -fpr at least 5 minutes. Llf no such 5-minute 
< ,* . block of time is observed, Item 7 is rated N/A].) 



SIRS 



1 - When student is engaged in academic s^t^work activities, little 
attention"!' s gj.ven by. teacher who directs seat-work activities 
from a distance or engages in work separate from the assigned 
seat work/ Teacher is general^ not helpful or supportive to 
student during Independent practice time. 

8. Corrections 

5 - The student's errors are consistently corrected by the teacher." 
When the student either does not respond, responds incorrectly, 
or does not respond jn unison if the activity Is group directed 
and requires such r^aponding, the teacher will systematically 
- attempt to correct the student by asking a simpler question, re- 
focusing student's attention to elicit correct response from the 
student or provide general rules by which to determine the 
correct answer 90% or more of the time. 

1 - Student's errors are rarely and inconsistently corrected by the 
teacher. The student responses are not systematically corrected 
Student's errors are corrected 50% or less of the time. 



For example: In, oral reading this includes teacher correction of skips 
and. mispronunciations, or help in sounding out hesitations 

9 . Positive Consequences 

5 - Positive events (tokens, points, activities, etc.) are given to 
the student when performing the desired behavior. "When learning 
a new skill the student receives positive consequencfe for. 
approximations of the desired behavior. 'Consequences are con- 
sistently received during academic training time. : Praise and 
compliments, e.g., "good working, .jiice job," are not included 
in this definition 4 . 

• & 

1 - Student rarely receives positive consequences fx>r academic work. 
When student receives consequences they usually are for social 
. behavior, rather than for behaviors occurring under systematic 
academic training. 

10, Qicing * ' 

5 - The pace of the lesson is rapid, providing many opportunities 
* for 'response by the student. As a result, attentipn is high 
and off-task behavior is low. 

» 1 - The pace of the lesson is slow and the student's rate of 

responding is low. Lesson format frequently varies, is not 
highly structured, and student attention may.be low. 



• A- 5 

SIRS, 

1 1 . Oral Practice on Outcome Behavior 

5 - Student reads aloud from context nearly all the time (85-100$ 
or 12-15 mtn. of a 15 min. observation). 

' 1 - Student does not read aloud during the observation (0% of the 
time). < 

(Note: Reading aloud for measurement purposes should not be considered 
when rating this variable. Readingjn context is defined as 
reading phrases, sentences, paragraphs, or story selections.) 



Examples: \ * * , 

If the student is reading isolated words nearly the entire time, 
the appropriate rating is a 3. 

If the student is reading aloud from a text about half the time, 
a 3 would be recorded, 

12. » Silent Practice on Outcome Behavior * i 

5 - Student reads silently from context nearly all the time .(85-100%, 
. or 12-15 nun. of a 15 min. observation). 

1 - Student does not read silently during the observation (0% of 
the time). 

(Note: Reading in context is defined as the same as*#ll. The examples 
.of #11 are the same for #12, with silent reading.) 
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School 
Date: 



Appendix B . 
Accuracy of Implementation Rating Scale 

■ .« " * Student: 



Observer (Items 1 and 2,): 
Rater (Items 3-13): 



Teacher 
* 



Number of observations prior to rating^_| ' 

Time observation begins : Time observation ends : m 

Time allocated to reading instruction per day: . 

Curriculum used for measurement: Publisher ■ 

Series Level e 



Instructions 

Circle the number that accurately reflects your rating for each 
variable. Only one number may be circled per variable. 1 reflects a 
low level of implementation and 5 means total implementation of the 
Procedures to Develop and Monitor Progress oh IEP Goals, See Operation- 
al Definitions. Items 1 and 2 requfre direct observation of the measure- 
ment administration. Items 3, ,4 t 5, 6 f and 7 require inspection of the 
student graph. Items 8, 9, and 10 require inspection of the student's 
IERform. The Instructional Plan must be inspected to rate item . 
The Change Record* must be inspected to rate items 12 and 13. 

» *• 

1. Administering the Measurement Task 

2. Selecting the Stimulus Material 

3. - Sampling for Instructional Level 

4. Baseline N \ s 

5. Graph Set-up 

6. Aimline 

7. Timing of Instructional Changes ^ 

'8. Long-Range Goal 
9. Short-Term Objective 
ID. Measurement System 

|Tl . Instructional Plan 

|T2. Substantial Changes 
jl3. One, Clear Change 
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AIRS 

Operational Definitions 
« Accuracy o.f Implementation Rating Scale 
Administering the Measurement Task 
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The measurement task is administered correctly: teacher 
brings stopwatch and pencil to measurement area; gives 
correct directions for the task; administers the measure- 
ment protedure for one minute; correctly .marks the teacher 
copy; correctly counts words correct and incorrect; cor- 
rectly counts words correct and incorrect; correctly 
plots the data point. 
• . •** \ * • '* 

1 - The teacher: forgets necessary materials; does not give • 
directions; does not time the task accurately; fails to 
.mark the teacher copy or incorrectly marks errors; miscounts 
correct and incorrect words; and inaccurately plots the data 
. point. < , 

2. Selecting the Stimulus Material 

5 -*The teacher has followed these procedures: Uses passages r 
selected^from -the level that represents the annual goal . 
Observers should record the book from which the passage 
was selected and later* check this vn'th* the long-range goal 
level. At this level find the pages in these stories th^t 
do not have excessive dialogue, indentations, and/or unusual 
pronouns. Write these page numbers on equal size slips of 
paper. 

- Put the slips of paper into a drawbag and shake it. - ; 

- Randomly pick a slip of papert 

- The page number chosen is the page where the student 
begins reading. If the page chosen is a passage that 
was read earlier during the week, draw another page^ 
number s . 

r. ft 

Other completely random procedures are also rated a 5. If, 
however, not all passages have an equal chance of being 
selected, a 4 rating, would be indicated. 

1 - The teacher fails to randomly pick the passage or the sample is 
* taken frgm a domain which is greater or smaller than the one 
-indicated in the goal . • * - ' 

3. Sampling for Instructional Level 

5 - The teacher has sampled from higher or lower reading levels 
to fftid the level in which the student reads 20 7 29 wpm 
(grades 1 & 2) or 30-39 wpm (grades 3 and up). 

o 

A ^ 



1 - The teacher is measuring -at a level which is too high or 

v too low. , • 

^ * 

Basel ine r 

5 - The student's performance has been measured at least 3 times to 
establish a stable baseline, A stable baseline means that all 
data points fall within a range of 10. • 

1 - The teacher has not found a level for which a stable baseline 
has been established or has failed to' collect 3 data points ,. 
during the baseline phase. 

Graph Set-Up • • * - 

5 - The graph is accurately set up: The dates filled in. on the 
horizontal axis;, the vertical axis is correctly labeled words 
read per minute from material; the units of measure- 
ment are specified; the student's name and subject area are 
certified; a key identifies the symbols for corrgct (.) and 
incofrreft (x); symbols are placed at the intersection of date 
aijd score; the data points are connected with straight lines; 
and absences are recorded on the graph as (abs.). 

1 The graph does not include many of the items mentioned above. 
Aim! ine 

5 - The long-range goal is marked on the graph with an X at the 
intersection of the desired performance level and date of 
"attainment and a line of ddsired progress connects the 
point representing the student's median score of the last 
3 data points from baseline and the LRG. 

1 - The long-range goal is not marked on the graph and/or the 
med4a*L_and LRG are not connected. \ 

Timing of Instructional Changes 

5 - All the adjustments in the 1 student 's program are made at th^ 
I appropriate time given the rujes for data utilization: 

(1) Compare the actual slope based on 7 to 10 data points 
to the slope required to attain the Annual Goal. 

(2) If the actual slope fs equal to, or steeper than, the 
Annual Goal slope, continue the program. 

(3) If the actual slope is flatter than the Annual Goal 
slope, change the- program. 

1 - None of the adjustments in the student's program are made 
at^the appropriate time. 
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8. Long-Range Goal 

5 - The long-range goal is accurately written; goal specifies 

. the number of weeks until 'next review; stimulus materials \ 
for the goal represents the level in which the student 
is performing at entry level criterion; goal specifies 
student behavior; goal specifies mastery criterion of 
50 wpm with fewer than 5 errors (grades 1 & 2) or 70 wpm 
with fewer than 7 errors (grades 3-5) when there are 36 
weeks until the annual review. If there are fewer than 36 
weeks, the criteria can be lowered proportionately. 

1 r The long-range goal contains none of the above criteria. 

9. Short-Term Objective 

5 - The short-term. objective is accurately written ; stimulus 
material and behavior is specified; and the average increase 
in performance is the desired performance minus the actual 
performance divided by the number of weeks until the annual 
review. 

T - The short-term objective contains none of the above criteria. 

10. Measurement System , 

5 - The teacher has indicated how the material is organized, the 
/frequency of measurement, and what is to be recorded on the 
graph. 

1 - The measurement system is not specified. 

11. Instructional Plan 

5 - The instructional plan includes clear and specific descriptions 
of the instructional procedures, the time spent in each acti- 
vity, the peKinent materials, the arrangements, and the 
motivational strategies. 

1 - The instructional plan is unclear and lacks specific descrip- 
tions of the instructional procedures, the time spent in each 
activity, the pertinent materials, the arrangements, and the 
motivational strategies. ' 

* 12. Substantial Change s 

5 - The adjustments in the student's program are always substantial 
(have a good chance of being effective; see Unit XIV). 

1 - Jhe adjustments are never substantial . 
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Clear Change 

5 - All the adjustments made introduce only one, clear program 
change. 

1 - All the adjustments made introduce more than one change 
and/or the change is unclear. 
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